Add agentless Feature Flagging configuration source#11892
Add agentless Feature Flagging configuration source#11892leoromanovsky wants to merge 4 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
1858f32 to
82bb199
Compare
be3f8fc to
f9e70e6
Compare
| final StringBuilder endpoint = | ||
| new StringBuilder("https://api.") | ||
| .append(config.getSite().toLowerCase(Locale.ROOT)) | ||
| .append(DATADOG_API_SERVER_DISTRIBUTION_PATH); |
There was a problem hiding this comment.
🟡 Blocker: This is going to be a CDN-backed API. We are waiting on the DNS to be settled.
| // TODO before merge: confirm the final backend route with the server-distribution API owners. | ||
| private static final String DATADOG_API_SERVER_DISTRIBUTION_PATH = | ||
| "/api/v2/feature-flagging/config/server-distribution"; |
There was a problem hiding this comment.
🟡 Blocker: The path is not settled yet.
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
There was a problem hiding this comment.
All retry, ETag, shutdown, and idempotency paths look correct. One real gap: apply() silently returns false for 401/403 with no log above DEBUG — users with a wrong or missing DD_API_KEY have no way to diagnose why feature flags never load. Fixed by separating the auth-failure branch in apply() with a LOGGER.warn(...) call.
🤖 Datadog Autotest · Commit f9e70e6 · What is Autotest? · Any feedback? Reach out in #autotest
| if (response.status == HttpURLConnection.HTTP_UNAUTHORIZED | ||
| || response.status == HttpURLConnection.HTTP_FORBIDDEN | ||
| || response.status != HttpURLConnection.HTTP_OK | ||
| || response.body == null) { | ||
| return false; |
There was a problem hiding this comment.
401/403 auth failures logged at DEBUG only — invisible in production logs
Feature flags silently never load for every user with a misconfigured API key — flags fall back to defaults with no observable diagnostic in standard log output
Assertion details
- Input: Agentless poller issues a request to the UFC endpoint while DD_API_KEY is missing or invalid; server returns HTTP 401 or 403
- Expected:
A WARN-level log entry so operators can identify and fix the misconfiguration - Actual:
apply()returnsfalsesilently; the only LOGGER calls in the class are at DEBUG level, so the failure produces no output under the default log level
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · Any feedback? Reach out in #autotest
| if (response.status == HttpURLConnection.HTTP_UNAUTHORIZED | |
| || response.status == HttpURLConnection.HTTP_FORBIDDEN | |
| || response.status != HttpURLConnection.HTTP_OK | |
| || response.body == null) { | |
| return false; | |
| if (response.status == HttpURLConnection.HTTP_UNAUTHORIZED | |
| || response.status == HttpURLConnection.HTTP_FORBIDDEN) { | |
| LOGGER.warn( | |
| "Feature Flagging agentless endpoint returned {}; verify DD_API_KEY is set and valid", | |
| response.status); | |
| return false; | |
| } | |
| if (response.status != HttpURLConnection.HTTP_OK || response.body == null) { | |
| return false; | |
| } |
NOTE TO REVIEWERS
This PR intentionally keeps the complete feature in one PR so the final architecture remains visible. Its four commits are deliberately layered like a stacked PR, and each commit contains the final production behavior and tests for that layer.
How to Review
Commit Guide
LOC is rename-aware per commit against its parent.
b8bbd269e04baefdbf6e304, last-known-good, bounded jittered retry, no overlap, in-flight cancellation, shutdown safety, and system-test-parity tests.d378350cf1f9e70e67c4Per-Commit Validation
Each commit was checked out and validated independently before publication:
ConfigTest, feature-flagging lib/agent tests, and relevant Spotless checks.All four gates completed with
BUILD SUCCESSFUL.Stack Position
You are here: the Java source-mode foundation. It makes Datadog-managed agentless delivery the default, keeps custom HTTP under agentless, and preserves Agent Remote Configuration as explicit opt-in.
flowchart LR subgraph JAVA["dd-trace-java"] J1["JAVA-01 · #11892<br/>Agentless + RC sources"] --> J2["JAVA-02 · #11639<br/>Aggregate evaluation EVP"] end subgraph SYSTEM["system-tests"] ST1["ST-01 · #7298<br/>Mock agentless backend"] --> ST2["ST-02 · #7299<br/>Side-effect contracts"] ST2 --> STM1["ST-M01 · #7300<br/>Enable Java configuration"] ST2 --> NEXT["Next drafts<br/>Enable Java side effects"] end subgraph DOGFOOD["ffe-dogfooding"] DOG0["DOG-00 · #92<br/>Agentless evaluation baseline"] --> DOG1["DOG-01 · #93<br/>Side-effect conduit"] end J1 --> STM1 J1 --> DOG0 J2 --> NEXT J2 --> DOG1 STM1 --> GREEN["Java proof<br/>both sources × side effects"] NEXT --> GREEN DOG1 --> GREEN classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class J1 current;Motivation
Java Feature Flagging needs a tracer-side configuration-source split so SDKs can fetch UFC from an agentless HTTP backend while preserving the existing Agent Remote Configuration path.
Changes
DD_FEATURE_FLAGS_CONFIGURATION_SOURCEwithagentless,remote_config, and reservedoffline; default isagentless.DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL.DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDSandDD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS.AgentlessConfigurationSource, an HTTP UFC poller used for both the default Datadog-managed endpoint and a configured custom HTTP endpoint.remote_configon the existingRemoteConfigServiceImpl/ Agent RC path.DD-API-KEY, accepted-200-only ETags/304, malformed UFC rejection, last-known-good preservation, and no overlapping polls.clamp(P/6, 2s, 10s)thenclamp(P/3, 5s, 30s).DD_FEATURE_FLAGS_ENABLEDand extra-header env from this PR.Decisions
No new provider kill switch. This keeps the existing provider bootstrap behavior through
DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED; changing enablement is out of scope for this PR.Custom HTTP delivery remains under
agentless. The source mode describes direct HTTP UFC delivery without the Datadog Agent. With no base URL, the SDK derives the first-party Datadog endpoint ashttps://api.<site>/api/v2/feature-flagging/config/server-distribution?dd_env=<env>. SettingDD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URLchooses a system-test, dogfood, or operator-managed HTTP endpoint without introducing a fourth source mode. A bare host uses the standard server-distribution path; a URL with a path is used as the exact UFC endpoint.Remote Configuration remains a distinct mode. It is Agent-mediated and uses the RC protocol's security, signing, targeting, capability, and subscription lifecycle. Those semantics are materially different from direct HTTP polling, so
remote_configremains a peer source mode rather than another agentless endpoint choice.No offline factory yet.
offlineremains a reserved configuration-source value. This PR does not expose a startup-bytes API or offline factory because that API still needs design work.Source selection stays SDK-local.
DD_FEATURE_FLAGS_CONFIGURATION_SOURCEdecides which local source implementation starts. It is not sent to the backend.Initialization Modes
Solid connectors are implemented in this PR. The dashed offline connector previews startup-provided UFC bytes.
flowchart TD Gate{DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED} Disabled[Feature Flagging subsystem not started] System[FeatureFlaggingSystem.start] Select{DD_FEATURE_FLAGS_CONFIGURATION_SOURCE} Exposure[ExposureWriter starts for every enabled mode] subgraph AgentlessMode[agentless mode - current default] AgentlessSource[AgentlessConfigurationSource] OperationalDefaults[SDK-owned operation<br/>30s poll default; 2s request timeout default<br/>ETag; retry; last-known-good; no overlap] EndpointChoice{agentless base URL set?} DatadogManaged[Datadog-managed agentless<br/>first-party; CDN-backed] CustomHttp[Custom HTTP endpoint<br/>system tests; dogfood; operator backend] AgentlessSource --- OperationalDefaults AgentlessSource -- polls --> EndpointChoice EndpointChoice -- no --> DatadogManaged EndpointChoice -- yes --> CustomHttp end RemoteConfig[remote_config<br/>explicit opt-in; Agent-mediated] RemoteConfigSource[RemoteConfigServiceImpl] AgentRc[Datadog Agent Remote Configuration<br/>RC security; signing; targeting; subscriptions] OfflineReserved[offline today<br/>reserved; no configuration service] OfflineSource[Later: OfflineConfigurationSource<br/>customer UFC JSON bytes at startup; no network] Ufc[Shared UFC deserialize and evaluate pipeline] Gateway[FeatureFlaggingGateway] Provider[OpenFeature provider] Gate -- false --> Disabled Gate -- true --> System System --> Select System --> Exposure Select -- unset or agentless --> AgentlessSource DatadogManaged --> Ufc CustomHttp --> Ufc Select -- remote_config --> RemoteConfig RemoteConfig --> RemoteConfigSource RemoteConfigSource -- subscribes --> AgentRc RemoteConfigSource --> Ufc Select -- offline today --> OfflineReserved OfflineReserved -. later .-> OfflineSource OfflineSource -. startup UFC bytes .-> Ufc Ufc --> Gateway Gateway --> ProviderCustomer Usage Example
Default Datadog-managed agentless delivery:
Custom HTTP delivery keeps agentless mode and changes only the endpoint:
Explicit Agent Remote Configuration delivery still uses the existing Agent path:
Verification
System Test Evidence
I built
dd-java-agent,dd-trace-api, anddd-openfeaturefrom the rewrittenhead
f9e70e67c4, rebuilt the Java Spring Boot weblog image with those localartifacts, and verified the JAR inside the image matched the local agent JAR.
I then ran the Java manifest-enabled agentless scenario from the companion
system-tests draft:
This exercises the Java manifest row proposed by the companion draft against
the exact code currently published in this PR.
Next Steps
DataDog/system-tests#7300.
merge the activation only when that gate is green.